feat: add array_exists with lambda support via CometUDF framework by andygrove · Pull Request #4223 · apache/datafusion-comet

andygrove · 2026-05-05T08:39:33Z

Which issue does this PR close?

Part of #4193

Rationale for this change

The CometUDF framework merged in #4232 and #4306 lets Comet dispatch scalar UDFs to JVM-side implementations that operate directly on Arrow data, avoiding the cost of falling back to Spark. This PR is the first consumer: array_exists (Spark's exists(array, x -> predicate(x))) ships as a JVM UDF so we can support a lambda-based expression with 100% Spark compatibility without porting the lambda evaluator to native code.

Performance is 1.8x of Spark on a 7M-row int-array workload.

OpenJDK 64-Bit Server VM 17.0.10+7-LTS on Mac OS X 26.2
Apple M3 Max
array_exists - int array (x -> x < 0):    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------
Spark                                               584            591           7          7.2         139.3       1.0X
Comet (Scan)                                        588            623          40          7.1         140.1       1.0X
Comet (Scan + Exec)                                 322            329           3         13.0          76.8       1.8X

What changes are included in this PR?

Experimental support for Spark's exists(array, x -> predicate(x)): the first lambda-based expression accelerated by Comet, built on the CometUDF framework.

ArrayExistsUDF: iterates ListVector elements, evaluates the lambda predicate via Spark's NamedLambdaVariable, implements three-valued null logic
CometArrayExists serde: registers the lambda in CometLambdaRegistry and emits a JvmScalarUdf proto invoking ArrayExistsUDF
Scope: single-argument lambdas that reference only the lambda variable; element types boolean, byte, short, int, long, float, double, decimal, date, timestamp, string

How are these changes tested?

CometArrayExpressionSuite covers compatibility and fallback paths.

Compatibility (checkSparkAnswerAndOperator):

integer and string predicates
null elements with three-valued logic
all-elements-match and empty-array cases
DataFrame API
decimal, date, timestamp element types
literal-only lambda bodies (true, false, null)
CASE WHEN and IF inside the lambda body

Fallback (checkSparkAnswerAndFallbackReason):

binary element type
lambdas referencing outer columns
nested lambdas

andygrove · 2026-05-05T15:06:21Z

@hsiang-c fyi

Adds a new JVM UDF bridge framework that allows Spark expressions to be evaluated on the JVM side via Arrow C Data Interface, while keeping the native execution pipeline intact. Includes array_exists as the first lambda-based expression using this framework.

comphead · 2026-05-05T17:00:22Z

Are we planning to merge it asap or wait DF 54.0?

comphead · 2026-05-05T17:02:40Z

we can try use apache/datafusion#21903 directly or create datafusion-spark counterparty

andygrove · 2026-05-05T18:44:11Z

Are we planning to merge it asap or wait DF 54.0?

I would love to get the JVM UDF framework in (once reviewed).

There are many applications where it can help us get acceleration by default rather than opt-in

lambdas
JSON
regex

What would be the advantage of waiting for DF 54? Does that give us 100% compatibility for array_exists with lambdas?

andygrove · 2026-05-05T19:04:32Z

Are we planning to merge it asap or wait DF 54.0?

I would love to get the JVM UDF framework in (once reviewed).

There are many applications where it can help us get acceleration by default rather than opt-in

lambdas

JSON

regex

What would be the advantage of waiting for DF 54? Does that give us 100% compatibility for array_exists with lambdas?

I could split the JVM UDF work out into a separate PR but there would be no tests if we don't have an example of an expression using it

comphead · 2026-05-05T21:15:16Z

Are we planning to merge it asap or wait DF 54.0?

I would love to get the JVM UDF framework in (once reviewed).
There are many applications where it can help us get acceleration by default rather than opt-in

lambdas

JSON

regex

What would be the advantage of waiting for DF 54? Does that give us 100% compatibility for array_exists with lambdas?

I could split the JVM UDF work out into a separate PR but there would be no tests if we don't have an example of an expression using it

Having no tests for lambda is fine IMO as we do not expose the feature to users right away.

Does that give us 100% compatibility for array_exists with lambdas?

Thats the entire intention of datafusion-spark module, but the compatibility is not always true unfortunately.

My main concern we could end up with multiple lambda implementation in DF and in Comet and might cause confusion and conflicts. The small poc PR shown the array_exists works with basic lambda(no column capture, no nested lambdas) https://github.com/apache/datafusion-comet/pull/4127/changes#diff-7411f5845a2488bb1509b95d8ad1e014422e21d70e0b802bd7624eabc4621c66

For customers we can build another branch on top of DF54 migration branch and including lambda functions there, so they can test it, WDYT?

andygrove · 2026-05-05T21:28:46Z

Having no tests for lambda is fine IMO as we do not expose the feature to users right away.

Ok, here is new PR with just the framework - #4232

Moving this PR to draft

wForget · 2026-05-07T02:17:37Z

+      "ArrayExistsUDF requires a non-null scalar registry key")
+
+    val registryKey = new String(keyVec.get(0), StandardCharsets.UTF_8)
+    val arrayExistsExpr = CometLambdaRegistry.get(registryKey).asInstanceOf[ArrayExists]


CometLambdaRegistry.register is called on the driver side, while CometLambdaRegistry.get is called on the executor side. Can the expression be obtained correctly?

https://github.com/andygrove/datafusion-comet/blob/ba15abed6dfa702051178384050abc2857b8cbf6/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L855

Good catch. Thanks.

# Conflicts: # native/Cargo.lock # native/core/src/execution/planner.rs # native/jni-bridge/src/comet_udf_bridge.rs # native/spark-expr/src/jvm_udf/mod.rs

Followup to the apache/main merge: the framework's evaluate signature gained numRows in PR apache#4306, but the ArrayExistsUDF override was missed.

… paths Adapted from PR apache#3611: - DataFrame API, decimal, date, and timestamp element types - Literal-only lambda bodies (true / false / null) - CaseWhen / If in lambda body - Fallback for binary element type - Fallback for lambdas capturing outer columns - Fallback for nested lambda

…istry The previous CometLambdaRegistry approach registered the ArrayExists expression in a JVM-local ConcurrentHashMap on the driver and looked it up by UUID on the executor. In a real cluster the executor JVM never sees the driver's registry and the lookup fails. Local-mode tests masked this because driver and executor share the same JVM. Serialize the ArrayExists Catalyst expression to bytes in the serde layer and embed it in the proto as a BinaryType literal arg. The UDF deserializes on each invocation. CometLambdaRegistry is no longer referenced and is removed.

andygrove mentioned this pull request May 5, 2026

Support higher-order array functions via JVM UDF bridge #4224

Open

andygrove marked this pull request as ready for review May 5, 2026 15:06

andygrove requested a review from comphead May 5, 2026 15:06

andygrove force-pushed the array-exists-lambda branch from a57dd14 to f1ece6c Compare May 5, 2026 16:50

andygrove mentioned this pull request May 5, 2026

feat: Add Comet UDF framework, and accelerate RLike with 100% Spark compatibility #4170

Closed

style: fix import ordering in jni-bridge lib.rs

ba15abe

andygrove marked this pull request as draft May 5, 2026 21:28

andygrove mentioned this pull request May 6, 2026

feat: add JVM UDF framework for native execution #4232

Merged

wForget reviewed May 7, 2026

View reviewed changes

Merge remote-tracking branch 'apache/main' into array-exists-lambda

cc2e993

# Conflicts: # native/Cargo.lock # native/core/src/execution/planner.rs # native/jni-bridge/src/comet_udf_bridge.rs # native/spark-expr/src/jvm_udf/mod.rs

andygrove changed the title ~~feat: implement array_exists with lambda support via JVM UDF bridge~~ feat: add array_exists with lambda support via CometUDF framework May 20, 2026

andygrove added 3 commits May 20, 2026 14:37

fix: align ArrayExistsUDF.evaluate with CometUDF numRows parameter

b03eb57

Followup to the apache/main merge: the framework's evaluate signature gained numRows in PR apache#4306, but the ArrayExistsUDF override was missed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add array_exists with lambda support via CometUDF framework#4223

feat: add array_exists with lambda support via CometUDF framework#4223
andygrove wants to merge 6 commits into
apache:mainfrom
andygrove:array-exists-lambda

andygrove commented May 5, 2026 •

edited

Loading

Uh oh!

andygrove commented May 5, 2026

Uh oh!

comphead commented May 5, 2026

Uh oh!

comphead commented May 5, 2026

Uh oh!

andygrove commented May 5, 2026

Uh oh!

andygrove commented May 5, 2026

Uh oh!

comphead commented May 5, 2026 •

edited

Loading

Uh oh!

andygrove commented May 5, 2026

Uh oh!

wForget May 7, 2026

Uh oh!

andygrove May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

andygrove commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

andygrove commented May 5, 2026

Uh oh!

comphead commented May 5, 2026

Uh oh!

comphead commented May 5, 2026

Uh oh!

andygrove commented May 5, 2026

Uh oh!

andygrove commented May 5, 2026

Uh oh!

comphead commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andygrove commented May 5, 2026

Uh oh!

wForget May 7, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andygrove commented May 5, 2026 •

edited

Loading

comphead commented May 5, 2026 •

edited

Loading